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Traditonally,  computer  systems  have  been  evaluated  in  terms  of  the 
cost/benefit  analysis  used  for  most  capital  investment  nroiects. 
Estimates  are  made  of  development  and  ooeratina  costs  and  of  savinas 
in  personnel,  equipment,  etc.   Apart  from  the  all  too  frequent  under- 
statement of  costs,  this  use  of  'hard'  measures  is  increasingly  diffi- 
cult to  apply  as  computer  systems  become  more  and  more  concerned  with 
aiding  the  management  decision  process  rather  than  with  automating 
existing  procedures.   A  time-shared  marketing  retrieval  system  provides 
'better'  information  and  implicitly  leads  to  'better'  decision-making.  These 

are   qualitative  benefits  which  are  hard  to  translate  into  dollar 
figures.   Unfortunately  these  systems  are  often  sold  to  top  management 
on  the  basis  of  such  hard  benefits  as  reduction  in  management  workload, 
which  may  or  may  not  materialize.   The  soft  benefits  which  are  the 
real  impetus  for  development  of  the  system  are  then  either  treated  as 
a  side  issue  or  ignored  because  top  management,  not  unreasonably,  will 
reject  a  capital  investment  nroposal  presented  in  terms  of  such  vague 
payoffs  as  improved  decisions,  more  timelv  information,  etc.   '^his  can 
lead  to  a  hidden  agenda,  with  the  computer  personnel  and  management 
sponsor  formally  justifying  a  project  because  of  hard  navoffs  (therebv 
establishing  these  as  criteria  for  evaluating  tte  outcome)  while  planning 
the  system  more  in  relation  to  the  soft,  qualitative  factors  that  reallv 
concern  them. 

An  addit-rnal  difficulty  with  the  cost/benefit  approach  is  that 
evaluation  ger.  -rally  has  to  be  post  facto,  when  the  project  has  been 
completed.   At  -har  point  it  is  largely  irrelevant,  since  the  develop- 
ment expenses  .  a  men  a  sunk  cost.   The  system  may  by  that  time  be 
viewed  as  effective  a.,  valuable  by  its  user  population  but  such 
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co.'.siderations  were  not  included  as  part  of  the  formal  justification 
of  the  venture.   It  is  hard  for  the  sponsor  of  the  project  to  argue  that 
in  fact  the  original  investment  of  real  dollars  has  been  adequately 
returned  thro'-.ch  these  intangible  benefits. 

The  evaluation  of  computer  aids  to  decision-making  is  a  complex 
process  and  it  is  ooviously  essential  that  the  problem  of  assessing 
qualitative  benefits  be  answered.   In  some  cases  it  is  possible  to 
convert  soft  factors  into  dollar  figures,  by  calculating  some  sort  of 
opportunity  cost  or  projecting  the  likely  impact  of  the  system  on  the 
organization's  competitive  position.   For  example,  the  user  group  may 
calculate  what   they  would  be  prepared  to  pay  some  outside  supplier 
for  the  information  provided  by  a  proposed  system  or  calculate  the 
likely  savings  and  improvements  in  last  year's  operations  had  the 
svst'^T"  >^<=pn  availa^lf^.   nbviouslv,  such  calculations  are  at  best 
approximations.   An  alternative  is  to  ignore  the  issue  and  accept 
that  computer  systems  are  largely  an  act  of  faith  -  or  a  marketing 
problem  in  which  top  r.anc-gement  must  be  sold  the  idea  that  an  investment 
of  $X  will  somehow  be  worth  it.   Phrases  such  as  'keeping  up  with  our 
competitors'  or  'maintaining  our  current  technological  position'  are 
very  useful  in  arguing  this  position.   The  third  alternative,  too 
conservative  to  be  desirable  except  in  the  short-term,  is  to  insist  that 
a.r.y  project  must  at  least  pay  off  in  terms  of  hard  benefits. 

All  these  approaches  may  be  adequate  in  particular  contexts.   They 
are  unlikely  to  remain  so,  however,  because  the  future  -  and  in  many 
instances  the  present  -  value  of  innovative  MIS  and  Management  Science 
ventures  are  mtendedly  and  entirely  qualitative.   The  following  example 
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may  already  be  typical: 

Megabuck  Inc. ,  a  large  financial  institution,  has  invested  well 
over  $1  million  in  a  complex  online  system  for  investment  management. 
This  uses  graphics  terminals,  draws  on  a  large  database  and  includes 
powerful  reporting  and  analytic  procedures.   It  is  currently  in  use  by 
a  group  of  30-40  senior  account  executives,  most  of  whom  earn  $30,000 
a  year.   The  system,  needless  to  say,  was  delivered  behind  schedule  and 
there  have  been  najor  hardware  problems,  some  of  which  have  been  only 
partially  solved.   The  system  is  significantly  different  from  the  original 
design.   User  reaction  has  been  mixed  and  there  is  still  some  resistance 
to  it. 

The  original  proposal  did  give  substantial  attention  to  qualitative 
benefits.   However,  the  main  justification  for  the  system  was  that  it 
would  lead  to  a  higher  rate  of  return  on  the  investment  portfolio  and 
would  enable  each  acco\int  executive  to  handle  more  clients. 

While  the  system  is  fully  operational,  additional  development  is 
needed,  particularly  to  speed  up  response  time  and  add  new  analytic 
routines.   The  operating  costs  for  the  system  are  over  $100,000  a  year. 

Top  management  is  reluctant  to  invest  any  more  money  and  has  asked 
for  a  full  evaluation  of  the  current  system.  The  staff  personnel  from 
Management  Science  group  who  are  responsible  for  the  evaluation  agree 
that  the  system  is  not  justified  in  terms  of  dollar  return  and  that  no 
real  case  can  be  made  -  on  a  cost/benefit  basis  -  for  spending  capital 
funds  on  further  development.  They  are  convinced,  as  is  the  line  man- 
agement in  the  investment  division,  that  the  system  is  critical  to  the 
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to  the  division  in  the  long-term.   They  have  been  unable  to  convince 
senior  management  of  this  and  recognize  that  quantifying  the  qualitative 
value  of  the  system  is  the  key  to  doing  so.   They  are  very  worried  and 
would  like  some  advice. 

The  rest  of  this  paper  constitutes  that  advice  -  too  late  unfortunately 
to  be  of  any  use  to  them,  since  it  starts  off  by  saying: 

"Evaluation  is  part  of  the  wider  process  of  implementation  and  begins 
before  the  system  is  designed. 

The  real  purpose  of  evaluation  should  not  be  a  post  facto  accounting, 
a  sort  of  score  card  on  what  was  accomplished  in  relation  to  what  was 
promised.   The  approach  to  evaluation  presented  in  the  rest  of  this  article 
is  essentially  diagnostic:   it  aims  at  embedding  evaluation  in  the 
implementation  process  so  that  the  central  question  of  soft  benefits 
is  dealt  with  from  the  onset  and  so  that  there  can  be  a  continuing  moni- 
toring of  the  system  during  its  development.   In  a  way,  the  ideas  pre- 
sented amount  to  a  model  of  implementation,  not  just  of  evaluation. 
It  is  interesting  to  note  that  there  has  been  a  subtle  but  accelerating 
shift  in  terminology  in  research  on  computer  systems-  only  a  few  years 
ago  the  dominant  word  in  titles  of  journal  papers  was  'Design'  while 
increasingly  that  has  been  supplanted  by  'Implementation'.   The  difference 
is  important  in  relation  to  evaluation.   The  emphasis  on  design  results 
in  evaluation  being  a  separate  stage  and  issue;  the  system  is  an  artefact 
which  is  later  assessed,  generally  on  the  same  basis  as  other  types  of 
capital  investment.   By  contrast,  the  emphasis  on  implementation  shifts 
the  focus  to  a  question  that  dominates  the  process  from  inception  to  finish: 
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What  are  we  trying  to  accomplish  and  how  will  we  recognize  when 
we  have  met  our  aims? 
That  question  can  be  answered  only  if: 

1)  the  initial  definition  of  the  system  aims  specifically  dealt  with  the 
issue  of  soft  benefits  and  identified  some  'key  indicators'  which 
can  be  used  to  measure  those  benefits 

2)  evaluation  is  fairly  continuous  so  that  we  can  measure,  even  if 
only  approximately,  where  we  are  in  relation  to  where  we  want  to  be 

3)  we  recognize  that  in  many  instances  the  final  system  will  be  substan- 
tially different  from  what  was  originally  planned;  modifications  to 
the  design,  including  the  further  evolution  of  an  operational  system 
such  as  that  described  in  the  Megabuck  example  above,  must  themselves 
be  evaluated  in  terms  of  their  contribution  to  the  wider  system  aims 

in  a  way,  the  dominant  problem  for  evaluation  is  simply  to  define 
what  is  the  'successful'  system.   There  is  a  tendency  to  assume  that 
'success'  is  equivalent  to  'use',  largely  because  usage  is  so  easily 
measured.   Jan  Huysmans ,  whose  paper  'Operations  Research  Implementation 
and  the  Practice  of  Management'   provides  a  rich  discussion  of  this  issue, 
defines  three 'levels  of  adoption'  of  OR  ventures  and  points  out  that 
these  may  overlap  or  even  be  in  conflict: 

1)  Does  the  development  of  an  OR  model  lead  to  a  management  action  ? 

2)  Does  the  development  of  an  OR  model  lead  to  a  management  change? 

3)  Does  management's  contact  with  the  OR  method  lead  to  recurring  use 
of  the  OR  approac'-'. .' 

One  might  add  to  Huysman's  list  in  the  context  of  computer  systems  as  a 
whole:   do    the  use  of  the  system  lead  to  redefinition  or  extension 
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of  the  task  it  supports? 

It  is  worth  noting  that  the  four  levels  of  adoption  (addinq  redefi- 
tion  of  task  as  the  fourth)  form  a  distinct  spectrum.   Level  1  (manage- 
ment action)  is  concerned  fairly  specificallv  with  the  model  or  svstem 
itself:  -the  key  questions  are: 

1)  Does  the  model  'solve'  the  problem? 

2)  Does  the  manager  act  on  the  basis  of  the  decision  (-identified  as  most 
desirable  by  the  model? 

3)  Is  the  model  technically  correct  -  dpes  it  provide  reliable  solutions? 
Level  4,  however,  is  much  less  concerned  with  a  sinale  svstemr  change 

in  task  will  require  change  in  management  behavior,  communications  and 

of  course  use  of  the  system.   The  time-frame  for  this  change  is  certain 

to  be  fairly  long,  and  the  process  of  chanae  evolutionary.   In  evaluatina 

a  system  which  aims  at  level  4,  one  must  assess  not  so  much  the  svstem 

as  its  impact.   Technical  elegance,  quality  of  solutions,  etc. ,  critical 

for  level  1,  are  much  less  relevant  for  level  4.   In  fact,  it  may  even  be 

that  a  system  that  aims  at  level  4  should  be  deliberately  simple  to  the 

point  of  being  inadequate  in  terms  of  level  1.   Rather  than  provide  a 

detailed  analysis  of  the  Droblem,  the  model  should  allov/  the  user  to  test  out 

broad  assumptions  and  a  wide  range  of  highly  varied  alternatives.   He 

can  then  define  his  problem  more  clearly  and  even  redefine  it,  maintainina 

a  broad  perspective.   For  example,  a  level  1  model  designed  to  help  a 

manager  schedule  production  for  a  six  month  period  might  focus  in  detail 

on  optimizing  cost  and  capacity  utilization.   A  level  4  model,  bv  contrast 

might  provide  less  feedback  on  costs  but  include  brief  analyses  of  pro-Fits 

and  inventory  posit-ons  assuming  particular  sales  forecasts.   In  this 

way  it  may  leac  the  user  to  view  the  production  function  from  a  companv-wide 


perspec-ive. 

Glen  Urban' s  Family  Plannina  System   provides  an  interestinq 
example  of  how  a  'poor'  model  can  be  of  great  value  to  a  manager.   The 
system  that  Urban  initially  designed,  Mod  1,  was  used  as  the  basis  for 
the  users  to  articulate  and  help  develop  a  larger  model  which  more  accu- 
rately captured  the  environment  it  represented.   Mod  1  is  a  poor  model 
in  the  sense  that  it  is  approximate,  general  and  missing  some  relevant 
data  and  variables.   However,  it  is  still  of  great  value;  it  is  an  aid 
in  'problem-finding'  and  is  helpful  to  inexperienced  users  both  as  an  intro- 
duction to  the  system  and  as  a  means  of  buildina  concepts  and  insight 
-  Mod  1  has  helped  its  users  rethink  and  refocus  their  decision  process 
in  a  way  that  probably  the  more  sophisticated  Mod  2  by  itself  would  not. 
There  are  many  other  situations  like  this  where  a  simple  model  that  its 
designer  may  disdain  can  be  of  immense  value  to  a  manager  and  is  thus 
a  ' success' . 

Obviously,  the  three  types  of  'success'  that  Huysmans  describes 
are  contingent,  that  is  there  are  many  instances  where  each  is  not 
applicable  and  may  even  be  undesireable.   Evaluation  starts  from  a 
definition  of  success  for  the  system  under  study;  that  definition 
obviously  has  to  be  provided  before  the  system  is  even  designed. 
It  is  clear  that  only  levels  1  and  2  in  the  list  above  can  be  assessed 
in  terms  of  use  of  the  system.   The  problem  then  is  how  to  locate 
adequate  indicators  for  the  other  levels.   The  list  below  expands  the 
four  levels  of  adoption,  suggesting  indicators  of  success  for  each  level; 
it  is  cursory  but  highlights  a  key  argument  of  this  paper,  that  success 
is  contingent  and  that  evaluation  similarly  should  be  contingent,  starting 
from  a  criterion  for  success  suitable  for  the  intended  level  of  adoption- 
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1)  does  the  system  generate  the  'solution'  manaqement  implements? 

2)  what  is  the  quality  of  the  system? (Little' s  criteria  of  simnlicitv, 

robustness,  ease  of  control,  adapt ivity,  completeness  on  imnortant 
issues,  ease  of  communication  between  user  and  system) 
Level  2  Management  change: 

1)  does  the  system  lead  management  to  identify  new  oroblems  or  new  ways 

of  dealing  with  existing  ones? 

2)  do  the  managers  initiate  follow-on  developments  as  a  result  of  this 

system? 

3)  in  similar  situations  as  the  one  which  led  to  the  implementation  of 

this  system,  do  the  managers  fairlv  automatically  assume  that  thev 
should  use  the  system  or  one  like  it? 
Level  3  Recurring  use  of  the  analytic  aoproach; 

1)  does  the  system  seem  to  have  led  to  managers  initiating  more  reauests 

for  analytic  aids? 

2)  do  they  request  education  in  analytic  methods? 

3)  does  the  analytic  approach  seem  to  be  more  integrated  into  their 

general  problem-solving  processes? 
Level  4  Change  in  task: 

1)  has  the  system  led  to  different  types  of  decisions  being  made? 

2)  has  the  control,  planning  and  performance  appraisal  systems  changed 

so  that  thev  now  include  factors  highlighted  by  the  model? 

3)  has  it  led  to  changes  in  organization  structures  or  personnel? 

Cavid  Kolb  and  Alan  Frohman  have  developed  a  qeneral  model  of  the 

consul:;ing  process  that  seems  particularly  applicable  to  the  implementa- 

4 
ti'.'.  of  computer  systems.    This  'organizational  Development'  approach 

to  onsultinq  ~rgues  that  any  social  change  process  requires  working 


through  a  series  of  stages.   More  importantly/  the  central  'Action' 
stage  -  which  is  generally  regarded  as  equivalent  to  implementation  - 
is  in  some  ways  the  least  critical.   Action  must  be  preceded  by  the 
establishment  of  a  contract  for  the  venture  and  followed  by  ^valuation 
and  then  by  the  Termination  stage,  where  the  system  or  program  is 
linked  into  the  ongoing  organizational  activities  so  that  it  becomes 
self-sustaining. 

The  stages  of  the  Kolb-Frohman  model  are  shown  below: 

SCOUTING 

ENTRY 

DIAGNOSIS 

PLANNING 

ACTION 

EVALUATION 

TERMINATION 
The  model  is  well  worth  studying  in  itself,  apart  from  iust  in  relation 
to  evaluation.   The  crux  of  Kolb  and  ^rohman's  argument  is  that  one  cannot 
simply  design  and  then  produce  a  change  nrogram.   The  wider  political, 
social  and  task  contingencies  of  the  organization  are  all  relevant  to 
design.   Social  change  of  any  sort  requires  above  all  the  creation  of  a 
climate  for  change.   Kurt  Lewin  calls  this  'unfreezing'  .   Resistance 
to  change  generally  comes  because  the  unfreezing  stage  has  not  been  worked 
through;  change  has  been  mandated  and  it  has  been  presumed  that  Action 
will  lead  to  acceptance,  but  the  organizational  system  can  often  dampen 
and  ar3orb  that  change.   The  early  stages  in  the  Kolb-Frohman  model  orenare 
fcr  th.-^  change  process.   The  Scouting  stage  involves  matching  the  skills 
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of  the  consultant  with  the  needs  of  the  client-organization.   Entrv 
requires  that  he  ensure  his  legitimacy  within  the  organization  bei^ore 
he  begins  the  design  of  his  change  program  (in  the  context  of  computer 
systems,  this  suggests  that  the  specialist  makes  sure  he  builds  contacts 
and  credibility  with  the  user  group,  and  not  just  with  his  'client'. 
Frequently,  the  initiation  of  a  system  involves  senior  staff  or  line 
management  within  a  division;  the  clerical  or  management  personnel 
who  will  actually  use  the  system  are  not  included  and  are  thus  not 
'unfrozen' ) . 

The  Entry  stage  is  perhaps  most  critical  of  all  for  evaluation  of  a 
system  Jihough  it  occurs  well  before  it  is  even  designed  )  since  it  is 
at  this  point  that  a  contract  for  implementation  is  negotiated.   Perhaps 
it  is  naive  to  assume  that  negotiation  between  consultant  and  client  will 
always  be  open  and  complete;  each  may  have  separate  and  partiallv 
incompatible  aims.   However,  the  ideal  agenda  for  negotiation  at  the  entry 
stage  is: 

1)  to  define  'success' 

2)  to  allocate  resources  and  responsibilities 

3)  to  develop  methods  and  criteria  for  evaluation,  including  a  consensus 

as  to  what  'key  indicator'  may  be  used  to  test  the  status  or 
accomplishment  of  each  aim  of  the  system. 

Megabuck's  investment  management  svstem  mav  be  used  as  an  example 
for  the  negotiation  process.   The  problem  Megabuck  faces  now  in  evaluating 
its  present  system  is  that  the  aims  were  never  pinned  down,   '^he  svstem 
v;as  intended  to  improve  the  qualityof  analysis  of  the  investment  managers. 
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lt was  also  assumed  that  it  would  lead  to  a  reduction  in  the  number  of 
new  investment  managers  to  be  hired  as  volume  of  business  increased. 
An  obvious  question  -  now  -  is  what  is  the  tradeoff  between  these  two 
aims?   It  seems  that  the  second,  reducing  personnel  needs,  has  been 
reasonably  met.  The  first  has  been  only  partiallv  accomplished.   If 
the  definition  for  success  had  been  'reduce  operating  costs'  then  the 
system  has  accomplished  what  was  intended.   The  original  designer 
of  the  system,  however,  was  much  more  concerned  with  building  in  the 
user  a  new  understanding  of  his  task.   He  hoped  that  the  investment 
managers  would  develop  a  wider  awareness  of  the  portfolio  as  a  whole 
rather  than  focus  on  each  individual  investment  as  a  separate 
consideration.   While  he  talked  about  this  aim,  it  was  never  included 
in  the  formal  specifications.   His  own  definition  of  success  was  fairly 
complex: 

1)  the  system  would  lead  to  a  redefinition  of  the  task  in  the  user's 

own  mind  which  would  lead  him  to  a  very  different  approach  to 
his  decision  making 

2)  it  would  lead  to  new  uses  of  existing  information;  the  organization 

research  personnel  would  have  a  more  direct  link  into  and  commu- 
nication with  the  user  group. 

3)  the  system  would  not  necessarily  be  heavily  or  continuouslv  used: 

depths  of  analysis  would  replace  breadth. 
By  those  criteria  the  present  system  is  inadeguate.   More  importantly, 
the  decision  as  to  whether  and  how  the  system  can  be  extended  may  well 
rest  3n  this  point;  the  designer  feels  that  a  relatively  small  expenditure 
on  certain  new  routines  will  immensely  help  the  managers  to  use  the  system 
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as  he  intended.   Ideally,  he  should  have  brought  his  concept  of  the  svstem 
more  into  the  open  at  the  start.   The  question  that  needed  to  be  asked 
is  conceptually  very  simple  'What  do  we  exoect  the  system  to  accomplish?' 
The  initial  answers  may  only  be  a  ].ist: 

1)  we  want  it  to  have  an  effect  over  the  lonq-term  on  how  our  managers 

actually  make  decisions;  we  want  them  to  think  much  more  in  terms 
of  formal  financial  theory,   oaying  attention  to  risk  considerations 
in  the  portfolio  as  a  whole. 

2)  we  want  it  to  help  the  expensive  research  staff  unit  have  more  impact 

on  the  managers  who  currently  treat  their  recommendations  as  advice 
that  they  may  (and  all  too  often  do)  ignore. 

3)  we  would  like  it  to  provide  this  without  dislocating  current  operations 

or  involving  substantial  expense  on  training. 

4)  we  expect  it  to  result  in  managers  being  able  to  handle  more  accounts 

5)  we  are  not  willing  to  spend  more  than  $X  to  develop  the  system  or  $Y 

per  year  to  operate  it. 

This  is  a  very  brief  and  imprecise  list.   It  does,  however,  form  a 
starting  point  for  specifying  criteria  for  evaluation.   First,  the  tradeoffs 
among  the  aims  should  be  determined.   There  are  at  least  three  groups,  top 
management,  the  users  and  the  designer,  involved  in  the  negotiation,  and 
their  priorities  may  differ.   The  designer,  for  example,  is  willing  to 
tradeoff  savings  on  personnel  for  improvements  in  the  investment  decision 
process.   The  user  may  not  be.   It  is  too  easy  for  the  designer  not  to 
make  his  viewpoint  explicit  and  to  later  work  into  the  system  design  those 
features  that  he  really  cares  about.   That  may  be  an  effective  political 
tactic  but  it  precludes  the  sort  of  evaluation  that  is  the  topic  here. 


-13- 
By  the  end  of  the  negotiation  stage  the  interested  parties  need  to  have 
spelled  out  a  formal  consensus  on  'success'.   Having  done  so,  responsibilitv 
for  meeting  the  aims  thus  defined  can  be  assigned  and  the  resources  needed 
identified.   The  order  is  important  and  provides  the  frame  for  meanina^ul 
cosu/benefit  analysis.   By  first  laying  out  what  the  aim  of  the  svstem  is, 
the  'soft'  issues  can  be  brought  out  explicitly;  when  that  is  done,  costs 
can  be  predicted.   Clearly  there  is  then  likely  to  be  a  regrouping:  the 
next  guestion  to  be  resolved  concerns  tradeoffs  -  are  we  willing  to  Dav 
SX  to  improve  communications  between  the  research  unit  and  the  investment 
manager?   If  we  are,  then  that  very  decision  attaches  some  hard  dollar 
figures  to  the  soft  benefit.   The  process  of  specif ving  benefits  and 
determining  the  resources  required  (which  may  also  be  partiallv  so^^t , 
such  as  manaaement  commitment,  etc)  is  recursive.   Its  end  product  should 
be,  as  Kolb  and  Frohman  explicitly  arque,  a  forma]  contract  for  r.esiar., 
implementation,  evaluation  and  termination  (the  last  of  which  will  re 
discussed  later  and  separately) . 

Having  decided  what  success  is  and  the  tradeoffs  between  benefits 
and  costs,  the  dilema  is  how  to  recognize  that  success.   'Improving 

communication  between  research  and  investment'  is  a  worthv  aim 
but   needs   to   be  made   operational.    It  may  well 

be  that  communication  can  be  measured  in  terms  of  requests  made  bv  the 

investment  group  to  the  research  unit.   In  that  case,  there  is  a  simple 

and  easily  tracked  criterion  for  assessing  this  asnect  of  the  svstem. 

On  the  other  hand,  no  surrogate  measure  for  communication  mav  be  apparent. 

Some  means  of  inventing  a  criterion  needs  to  be  found  or  this  system  aim 

musr  either  be  dropped  or  some  resolution  reached.   That  resolution  mav 

even  be  a  recognition  that  while  the  aim  is  worthwhile  its  achievement 
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is  simply  impossible  to  identify  so  that  it  must  be  releqated  to  the 
position  of  a  side  issue.   An  indicator  might  be  created  bv  qettinq  the 
investment  managers  to  identify,  on  a  weekly  basis,  anv  information  from 
the  research  group  that  has  definetly  influenced  or  altered  a  decision. 
That  is  not  a  very  satisfactory  solution;  the  point  to  be  stressed  is 
that  some   indicator  has  to  be  located.   The  negotiation  stage  is  reallv 
the  only  one  in  which  all  parties  are  present  and  no  commitments  have 
yet  been  made,  so  that  it  provides  a  chance  to  build  a  consens\is,  in 
effect  requiring  the  parties  to  agree  on  what  thev  will  all  accept 
as  evidence  on  a  particular  aspect  of  the  system. 

Consensus  is  an  important  nart  of  this  negotiation  process.   It 
may  be  well  worth  formally  analyzing  any  differences  in  opinion  between 
the  designer-consultant,  the  user  and  the  client  group  on  all  the  air.? 
and  key  indicators  that  are  discussed.   In  general,  anv  lack  of  ror..-:r-.;er-.-"--5 
between  the  groups  is  a  signal;  if  for  instance,  the  client's  view  o^ 
the  importance  of  improving  communication  or  as  to  what  kev  indicator 
should  be  used  differs  from  the  designer's,  that  lack  of  congruence  is 
likely  to  have  an  impact  on  the  implementation  process  that  can  be  costlv. 
Rather  than  ingore  the  difference,  it  should  -  perhaps  even  must-  be 
resolved.   Once  it  has  been  resolved,  the  proiect  has  a  clear-cut  set 
of  aims  and  a  specific  set  of  indicators  that  may  be  used  in  evaluation. 
All  the  hidden   agenda  will  then  have  been  brought  into  the  oper. . 

The  indicators  are  linked  to  the  initial  system  aims.   Because  of 
this  they  may  be  used  for  tracking  implementation.   A  maior  problem  in 
innovative  systems  is  to  decide  when  thev  are  complete.   In  the  Megabuck 
example,  several  of  the  users  in  fact  have  stated  that  the  promised  svstem 
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has  not  yet  been  delivered.   The  Kolb-Frohman  model  includes  a  final 
stage  called  Termination.   This  argues  that  the  consultant  needs  to 
include  in  the  contract  hammered  out  in  the  entry  stage  the  terms  of 
disengagement.   Many  change  programs  collapse  when  the  consultant  leaves 
the  organization.   Many  computer  systems  sink  into  oblivion  when  the 
designer  or  programmer  is  no  longer  around.   Fvaliiation  seems  esoeciallv 
useful  in  fine-tuning  an  'operational'  system,  determining  via  the  kev 
indicators  which  of  the  systems  aims  have  not  yet  been  adequately  reac^-ied 
and  providing  a  basis  for  deciding  when  the  system  is  comnlete  and  self- 
sustiining,  so  that  the  computer  personnel  can  hand  it  over  to  the  user 
groun.   'Generally,  the  terms  of  disengagement  will  include  such  issues 
as  training,  assigning  a  'manager'  for  the  system,  etc. 

The  three  central  stages  in  the  Kolb-Frohman  model,  Diaanosis, 
Planning  and  Action,  constitute  the  design  and  delivery  steps  for 
computer  systems.   Again,  it  is  worth  stressing  that  each  of  these  is 
very  dependent  on  the  contract  developed  in  Entrv.   It  constitutes 
benchmarks  for  design  and  for  making  the  decision  on  when  the  system 
is  complete.   The  evaluation  process  is  then  ongoing.   From  the  contract 
we  know  where  we  aim  to  go;  at  anv  point  in  the  implementation  process 
we  need  to  ask  where  we  are  now.   Doing  this  improves  the  likelihood 
that  the  system  will  be  treated  as  what  it  is,  a  change   program  that 
impacts  many  aspects  of  the  organi?ation,  not  iust  a  model  or  set  of 
programs.   'Where  are  we?'  is  the  central  question  f^roughout  implemen- 
tation; answering  it  gives  pointers  for  tactical  decisions,  such  as 
modifying  previously  promised  user  functions  because  of  cost  or  software 
problems.   This  style  of  intermediate  evaluation  oets  away  from  post 
facto  accounting.   It  makes  evaluation  not  iust  a  score  card  but  an  ongoina 
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diagnosis  and  tracking  that  may  also  be  of  help  in  dealing  with 
another  of  the  traditional  problems  in  implementation,  progress 
reporting  amd  project  management. 

The  approach  to  evaluation  outlined  here  is  a  state  of  mind  as 
much  as  a  technique.   One  simple  lesson  it  implies  is  'don't  rush'. 
There  is  a  tendency  among  computer  specialists  to  design  the  svstem 
and  get  it  up  and  running  and  then  deal  with  the  'people'  problems, 
politics  etc.   Megabuck  essentially  tried  this.   ^^he  svstem  had  both 
hardware  and  software  problems  and  was  behind  schedule  when  delivered. 
Only  then  did  the  oraanization  ask  what  had  been  accomplished  and  what 
it  was  worth.   The  issues  that  came  up  in  the  post  facto  evaluation  are 
all  ones  that  existed  and  should  have  been  raised  much,  much  earlier. 
In  particular,  a  change  in  the  investment  manager's  task  was  a  critical 
consideration  in  the  design  of  the  system,  but  never  got  resolved.   It 
was  openly  discussed  only  among  the  technical  personnel  in  the  user  and 
consulting  groups.   No  measure  of  success  was  deblded  upon  so  that  almost 
by  default  usage  was  adopted  as  the  indicator.   Usage  is  low,  which  of 
course  means  that  the  dollar  cost  per  session  or  hour  is  very  high. 
Several  outside  researchers  who  have  examined  the  Megabuck  svstem  feel 
that  it  i£  a  success  and  merits  continued  investment  and  evolution, 
because  it  is  shifting  the  users'  decision  process  toward  a  more  com- 
prehensive and  analytic  approach  that  seems  better  suited  to  the 
environment  Megabuck  will  face  over  the  next  decade.   However,  after  the 
event,  the  evaluation  process  cannot  deal  with  these  issues  v;ithout  soun^'- 
ing  like  self-justif ication.   The  irony  is  that  the  designer  had  considered 
them  in  great  detail.   Had  the  pre-design  step  included  an  explicit  dis- 
cussion and  operationalization  of  these  soft  aims  then  Megabuck  might  not 
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face  its  present  dilema  where  there  is  a  more  than  even  chance  that  the 
system  will  be  scrapped. 

The  argument  in  this  paper  has  concentrated  on  iustifyinq  an  approach 
to  evalaution  rather  than  spelling  out  the  technique  it  implies.   To  an 
extent,  the  approach  cannot  be  made  too  rigid  or  nrecise,  since  one  o-P 
its  central  aims  is  to  make  sure  that  the  difficult  and  elusive  soft  issues 
are  well  handled.   Each  project  will  require  a  different  wav  of  dealing 
with  them;  the  key  argument  of  this  paper  is  that  thev  must  He  dealt  with 
and  not  glossed  over.   Expandinq  the  approach  to  evaluation  discussed 
here  into  a  formal  technique  for  implementation  and  evaluation  clearlv 
requires  a  variety  of  adjustments  by  the  organization.  '^he   Kolb-Frohman 
model  in  particular  implies  that  'Organization  Pevelonment'  involves  the 
creation  of  a  climate  in  which  consensus  and  ooenness  are  encouraged  and 
conflict  resolved  rather  than  smoothed  over.   Glen  Urban,  whose  Famil" 
Planning  Svstem  was  mentioned  earler,  has  used  the  Kolb-^^rohman  approach 
as  the  basis  for  'Building  models  for  managers ' 6 .  His  experience  suqaests 
that  it  is  an  effective  strategy  for  introducing  computer  aids  to  decision- 
making into  the  organization.   Techniques  follow  concepts.   The  kev  aim 
in  this  paper  has  been  to  establish  some  concepts;  given  the  concepts 
many  techniques  suggest  themselves.   The  summary  below  provides  a  reasonably 
specific  framework  for  evaluation;  its  details  will  need  adaptina  to  the 
exact  conditions  of  a  particular  system: 
Et'JTRY  What  do  we  want  the  system  to  accomplish? 

1)  save  hard  dollars 

2)  change  the  wav  we  make  decisions 

3)  improve  communication 

4)  improve  information  flows 
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5)  be  easily  used 

6)  be  quickly  responsive 

7)  be  sophisticated  in  terms  of  routines 
Wliat  exactly  do  we  mean  hv  each  o^"  these? 

IVhft  priorities  do  we  assign  to  these  aims? 

What  key  indicators  will  we  use  to  evaluated  the  success 

of  each  aim? 
T-fhat  resources  are  required? 
V^o  is  responsible  for  each  resource  and  aim? 

DESIGN  Does  the  design  contribute  to  each  o^   the  aims^ 

Are  the  priorities  reflected  in  the  desicm? 

ACTION  As  the  system  is  brought  onstream,  track  the  extent 

to  which  each  aim  is  being  met,  via  the  kev  indicators. 

EVALUATION         Is  the  system  complete?: 

1)  have  the  predefined  aims  been  met? 

2)  do  the  kev  indicators  show  that  the  svstem 

is  generating  the  changes  and  benefits  expected? 

3)  what  are  the  costs  in  dollars  and  time  rerruired 

to  reach  ' success '-are  thev  worth  paving? 

4)  is  the  system  self-sustaining: 

do  the  users  need  training? 

is  the  system  compatible  with  the  organization's 

structure  and  procedures? 
is  the  consultant  now  redundant? 

If  the  system  evaluation  shows  it  is  not  complete  in  terms 

of  the  questions  above,  it  may  be  necessary  to  cvcle  back 

through  Design  and  Action. 


TERMINATION        The  project  is  ready  for  the  consultant  to  disenaaqe  when- 

1)  evaluation  shows  that  the  system  is 

'successful'  or 

2)  the  evaluation  shows  it  is  incomplete  hut 

there  is  consensus  among  the  interested 
parties  that  the  marginal  costs  of  the  extra 
effort  needed  is  not  justified 

Of  course,  the  set  of  statements  above  evade  many  realitites  of  real 
experience.   The  implementation  process  is  rarely  clear  cut,  many  o^.   the 
benefits  as  well  as  the  problems  are  unanticipated  and  the  organization 
itself  may  redefine  its  needs  as  it  learns  from  the  svstem.   Finallv, 
consensus  and  openness  are  not  easy  to  get;  they  reouire  a  climate  of 
behavior  that  is  widely  recommended  and  rarely  seen.   '^he  suggestions 
given  here,  however,  are  if  anything  strengthened  hv  these  cold  facts 
of  life.   If  implementation  is  bedeviled  bv  hardware  problems  or  the 
design  has  to  be  modified  for  any  reason,  the  process  will  lose  impetus 
and  direction  unless  at  the  very  start  'success'  was  defined.   Serendipitv 
is  a  welcome  outcome  of  a  system,  but  it  can  only  be  exploited  and  followed 
up  if  there  is  a  well-defined  direction  for  the  svstem  to  aim  towards. 
Whatever  the  cost  of  consensus,  it  is  important  to  recognize  that  without 
it  a  computer  development  project  runs  many  risks  and  may  move  from  the 
safety  of  the  technical  world  to  the  less  controllable  political  arena. 
A  designer  or  analyst  may  choose  to  wprk  with  a  hidden  agenda  but  if  he 
does  not,  then  it  is  in  his  and  the  project's  interest  that  he  encouraae 
others  not  to  do  so. 

Much  of  the  argument  presented  here  is  extremely  simple;  it  mainlv 
suggests  that  a  shift  in  focus  can  extend  the  evaluation  process  so  that 
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it  becomes  an  integral  part  of  implementation.   For  Meqabuck's  manaqe- 
ment  science  group,  evaluation  of  their  system  is  a  threat:  it  does  not 
deal  with  the  factors  that  really  concern  them  and  it  is  too  late.   E-.-5lu£ 
has  to  be  linked  into  the  goal-setting  process  and  it  must  above  all  ieal 
with  the  question  of  qualitative  issues;  if  those  are  made  the  starting 
point,  evaluation  is  likely  to  be  simpler,  more  effective  and  more 
relevant. 
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